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Abstract.  The  well  known  algorithm  of  David  A.  llnITman  for  finding  miniiniim  redundancy  codi's  has  fo\ind 
many  diverse  applications,  and  in  recent  years  it  has  been  extended  in  a  variety  of  ways,  'I'he  purpose  of  this 
note  is  to  discuss  a  simple  algebraic  approach  that  seems  to  fit  essentially  all  of  the  applications  of  lIulTnian's 
method  that  arc  presently  known. 
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Iluflinan’s  Algorithm  via  Algebra 


The  well  known  algorithm  of  David  A.  HufTinan  (5|  for  finding  minimum  redundancy  codes  has  found 
many  diverse  applications,  and  in  recent  years  it  has  been  extended  in  a  variety  of  ways  [1 ,2, 3, 4,6,7, 8j. 
The  purpose  of  this  note  is  to  discuss  a  simple  algebraic  approach  that  seems  to  lit  essiuitially  all  of  the 
applications  of  HulTinan’s  method  that  are  presently  known.  'I'lie  ideas  to  be  presented  are  slight  extensions 
and  corrections  of  the  results  in  [8). 

We  consider  a  linearly  ordered  set  A  on  which  a  binary  operator  has  been  defined  satisfying  the  following 
live  axioms; 

AO  (Increasing  property).  o  <  aob. 

A1  (Commutative  law).  aob  -=  baa. 

A2  (Cousin  law).  (ao6)o(co(i)  (a  a  c)^  (bod). 

A3  (Preservation  of  order).  If  a  <  6  then  no  c  <  bo  c. 

At  (A.ssocialive  inequality).  If  a  <  c  then  (n  6)  o  c  <  a  o  (6  o  c). 

Given  element.  a|,...,a„  f  A,  not  necessarily  distinct,  an  ex/>res.sion  on  (ui,  ..,a„)  is  a  formula  that 
computes  another  element  of  .1  he  applying  the  binary  operation  n  1  times  and  using  each  a,  exactly 
once.  For  example,  the  commutative  law  Al  slates  that  both  of  the  possible  expressions  on  have  the 

same  value,  and  the  cousin  law  A2  slates  that  two  particular  expressions  on  (n,  b,  r,d}  are  eipial,  Fxpressions 
are  essentially  binary  trees  having  {(i|,...,a„}  as  “leaves”. 

llulTinan's  algorithm  forms  an  expression  on  {«i,...,o„}  in  the  following  way:  If  ri  '  1,  let  a.  and  aj 
be  the  smallest  and  second-smallest  elements;  replace  a,  and  a  by  (a,  on  )  and  repeat  the  constriicljon  on 
the  remaining  n  1  elements,  until  eventually  n  1. 

For  example,  suppose  A  consists  of  the  nonnegalive  integers,  and  let  a  h  2(a  t  6).  It  is  ea.sy  to 
check  that  axioms  AO  A'l  hold.  IliilTman’s  algorithm  applied  to  the  elements  (1,3,  5,7,9}  will  produce  the 
expression 

((5o7)o((l  n3)o9))  116. 

Note  that  we  have 

5  2-  I  7-2^  1  ■•.>’  I  ,3  2-’  1  9  2-  116; 

III  the  casi'  of  this  particular  operation  the  value  is  ^o,  2'' ,  where  I  is  the  “level"  .it  winch  <i  appears  in  the 
furiiiiila,  i.e  ,  the  depth  of  parenthesis  nesting  when  parenlhe.ses  surround  each  use  of  the  binary  operation. 

When  a  binary  operator  salislies  the  coniniutalive  and  cousin  laws  .V  1  and  .X'J,  the  valui'  o.  any  expression 
on  (oi,.  ,a„}  depends  only  on  the  a,  and  their  U-vels  I,,  m  other  winds,  any  two  expressions  m  which 

each  a,  appears  on  a  given  level  I  will  be  equal  We  can  prove  this  by  using  terininologv  from  family  trees; 
If  ‘(a  e  A)'  appi'ars  in  some  formula  we  ran  say  llial  a  and  l>  are  brothers  and  (n  h)  is  their  father  Two 
eleiiietils  n,  and  n,  on  the  same  level  in  some  expression  are  either  hrolhers,  or  lin  y  are  cousins  (their  fathers 
are  brothers),  or  they  are  si'cond  cousins  (their  fathers  an-  cousins),  etc  We  can  Iraiisforiii  the  expression 
to  all  eipiivalenl  one  using  At  and  A2  iiiilil  a  and  «,  are  iirolhers;  for  if  a  and  n  ;ire  »f-l|i  cousins  and 
k  1,  the  axioms  chuiige  cousins  with  brothers,  while  if  A:  -  I  the  traiisformal  ion  for  order  k  1  will 
make  their  fathers  into  brothers  and  one  more  step  will  eomph  le  the  job  Now  let  /•.'  and  /•''  be  expr-ssions 
III  {(J|,  }  for  which  the  levels  I,  I'  agree  for  all  i  If  n  1  ,  clearly  /'.'  Otherwi-.e  h'  contains 

some  operation  (a,  on,).  Since  f,  f,,  we  ran  transform  li'  to  an  expression  in  which  u,  and  a. 

This  paper  is  dedicated  to  Marshall  Hall,  Jr  ,  on  the  occasion  of  Ins  ri-lnenii-iii  from  I'  u  liiiig. 


are  brothers.  Replacing  (a,  ooj)  by  a  new  symbol  in  E  and  E"  yields  an  expression  in  n  --  1  elements 
having  corresponding  level  numbers  equal,  hence  E  =  E" . 

The  main  feature  of  HufTman's  algorithm  is  that  it  produces  an  expression  of  minimum  value,  from 
among  all  expressions  on  the  given  elements  {oi, . . .  ,a„},  whenever  axioms  AO-A'l  hold.  We  can  prove  this 
by  starting  with  any  expression  E  and  transforming  it  into  expressions  of  equal  or  lesser  value  until  we  obtain 
the  result  of  Huffman’s  construction.  First  we  Jet  a,  be  tJic  smallest  of  [oi , . . . ,  a,,}.  If  /,  <  max(/|, . . . , 
(i.e.,  if  o,  is  not  at  the  deepest  level  of  E),  let  Ojt  be  an  element  at  the  deepest  level;  tlieti  some  ancestor  of 
Ok  is  at  the  same  le\jl  as  a,,  and  we  can  transform  E  into  E'  —  E  where  u,  is  a  brother  of  this  ancestor. 
One  of  the  nephews  of  a,  in  E'  is  or  an  ancestor  of  a/t;  call  it  x.  The  other  nephew,  call  it  y,  is  not. 
Since  y  has  been  computed  from  one  or  more  elements  greater  than  or  equal  to  a,,  we  have  a,  <  y  by  axiom 
AO.  Thus  (a,  oa:)oy  <  a,  o(ioy);  replacing  (a,  o(ioy))  by  ((a,  oi)oy)  in  /?'  yields  an  expression  E"  <  E' , 
because  of  axiom  A3.  Furthermore  a,  has  moved  to  a  deeper  level  in  E” ,  while  a*  is  still  at  the  .same  level, 
which  is  still  maximum  among  all  levels.  After  repeating  this  transformation  enough  times,  n,  will  appear 
at  the  deepest  level.  The  same  process  can  now  be  repeated  with  respect  to  the  second-smallest  element,  a,, 
this  time  using  a,  instead  of  in  the  argument.  Finally,  with  both  a,  and  on  the  same  level,  we  can 
make  them  brothers,  and  E  has  been  reduced  to  an  expression  E'"  contaii’ing  (a,  o  Oj).  Rirplacing  a,  and 
Oj  by  (a,  oaj),  we  can  repeat  the  process  until  the  de.sir  d  Iluffman-expression  has  been  reached. 

It  is  not  clear  that  axiom  AO  is  necessary  for  the  validity  of  this  result;  however,  lIulTman’s  construction 
leads  to  trees  of  comparatively  little  interest  if  axiom  AO  is  violated,  so  there  seems  to  be  little  harm  in 
assuming  AO.  It  can  be  shown  that  axioms  AO  -A'l  do  not  imply  the  law 

if  I  <  y  then  ((i  o  a)  o  6)  o  y  <  ((y  o  a)  o  h)  o  i  , 

although  this  seems  but  a  mild  extension  of  Ad.  Thus,  if  we  are  faced  with  an  expression  like  (((in6)c{ri'd))oe 
where  e  is  the  smallest  element,  wc  cannot  simply  exchange  e  with  d,  say,  in  an  attempt  to  move  e  to  the 
deepest  level;  the  argument  in  the  previous  paragraph  used  AO  to  conclinle  that  c  o  d  >  e,  so  that  c  could 
be  exchanged  with  (c  o  d)  via  A4. 

The  fact  that  Huffman’s  algorithm  produces  the  minimum  exprc.s.sioii  on  {n, , . . . ,  o,  }  does  not  obviously 
imply  Iluffman’s  original  theorem  that  the  minimum  value  of  ^  a, I,  is  obtained,  when  the  u,  ,-ire  nonnegativc 
real  rnttnbers  and  the  operation  a  o  6  is  simply  a  -f  6.  For  whenever  a  '  b  is  associative,  all  expressions  on 
{u), . .  .  ,a„}  .are  equal.  Previous  papers  about  abstr.actions  of  Huffman's  uielliod  have  tlu  refo"  worked  with 
two  separate  operations,  one  for  the  values  that  control  the  conslrnction  of  the  expression  and  the  other  for 
the  evaluation  function  that  is  to  be  minimized.  However,  it  is  possible  to  deduce  llulTinan’s  theorem  without 
this  extra  app.aratus,  by  defining  a  suitable  nonassoctative  operator  that  works  with  pairs  of  numbers  instead 
of  single  reals. 

Let  A  be  the  set  of  ordered  pairs  {a,  a')  of  nonnegative  real  numbers,  ordered  lexicographically  so  that 
(a, o')  <  (6,6')  if  a  <  6  or  a  6  and  a'  <  6'.  The  operation 

(a, o')  o  (6,6')  (n  1  6,  n  t  6  1  a'  I  6')  (1) 

is  easily  seen  to  satisfy  AO  Ad.  Therefore,  the  restill  of  Huffman’s  coiislriiclion  applied  U)  given  pairs 
{(a,,a',),...,{(o„,V„)}  is  an  expression  of  niiniinuin  value.  It  is  not  hard  to  see  that  the  value  of  any  such 
expression  with  respect  to  this  operator  is  the  pair 

(rti  -f  ' '  •  -f  ,  fl  1 1 1  “j-  •  • '  I  a„l„  d  a j  |.  . . .  [  a„ ) , 

where  each  1,  is  the  level  of  (n,,a',)  as  before.  Since  the  first  roint>onent  is  independent  of  the  ,  Iluffuiaii's 
construction  drms  indeed  miniinivic  ^o,/,. 
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Another  interesting  operation  on  pairs  is 


(a,  o')  o  (6,  b')  =  (max(a,  6)  -f-  1 ,  a'[ci  >  b)  )-  b'{b  >  a))  , 


(2) 


where  ‘(o  >  6)’  is  1  or  0  according  as  o  >  6  or  o  <  6.  This  operation  also  satisfies  AO-  A4;  for  example, 
when  verifying  A2  we  have 

{{a,  o')  o  (6, 6'))  o  ((c,  c')  o  (d,  d'))  = 

(max(a,  6,  c,  d)  -|  2  ,  a'(a  >  b,  c,  d)  b'{b  >  a,  c,  d)  -f  c'(c  >  a,  b,d)  |  d'(d  >  a,  b,  c))  , 

which  is  symmetrical  in  the  four  arguments.  Huffman’s  construction  produces  the  expression  of  minimal 
value,  which  in  this  case  is  the  minimum  value  of 


(  max  (Oj  +  ^  max  (oj  1  fj}). 

It  is  interesting  to  search  for  additional  operations  that  satisfy  AO  Ad,  since  each  of  'he.se  corro.sponds 
to  a  minimization  algorithm.  The  quadruple  operation 


(o,  o',  o",  o'")  o  (b.  6',  6",  6"')  =■-.  (a  f  b ,  a'  6' ,  o  |  6  |  o"  )  b"  ,  ,'  )  b'  | 


b'") 


illustrates  another  possibility;  the  optimum  in  this  case  is 


(3) 


l  EO, 

so  we  minimize  the  ‘weighted  path  length’  ^a,l,  and  -  among  all  trees  for  which  t/ii's  is  minimum  we 
minimize  another  weighted  path  length 

At  first  glance,  operations  (1),  (2),  (3)  may  seem  very  tricky  or  mysterious  or  both.  Actu.-.lly  there  is  a 
fairly  simple  way  to  account  for  all  of  them:  the  function 


a  o  b  x(a  b  b)  (4) 

satisfies  AO  A4  for  all  x  >  I,  over  the  nonnegative  reals.  Operation  (1)  corresponds  to  the  multiplier  x  - 
1  -b  (,  where  the  pairs  (a,  o')  correspond  to  polynomials  o  I  <n  (,  modulo  Operation  (3)  is  similar 
but  with  I  ~  1  t-  Operation  (2)  corresponds  to  large  values  of  x;  it  records  the  degree  and  loading 
coefficient  of  a  polynomial  in  x  so  that  (a, o')  <  ►  x"a'  (  0{i  "  ').  The  author  has  been  unable  to  conslrucl 

an  operation  that  "orresponds  to  minimization  of  max(f|, . . . ,  l„)  over  all  expressions  that  minimize  )Ja,f, ; 
it  seems  that  such  an  operation  might  exist,  pos.sibly  a  very  simple  one,  because  S'^hwartz  [')]  showed  that 
a  variant  of  HulTinan's  algorithm  does  this.  Tlie  arguments  of  I’arker  |8j,  that  isomorphic  versions  of  (4) 
account  for  essentially  all  operations  satisfying  AO  A4,  arc  incomplete,  but  it  doe  appear  that  all  of  the 
useful  operations  are  strongly  related  to  (4). 

When  aob  --  alb,  the  formvda  Et<i<,n^'^'  rewritten  Ei  <  j  < ..  •‘‘i  ■  where  {,sr , .  . .  ,  .s,,  i  }  is  the 

set  of  subexpressions  of  a  given  expression.  Kor  example,  in  the  expression  (({«|  |  n  .)  |  a  \)  |  (u)  I  n  ,)),  we 

haveSai  1  Sao  b2n;rl  2a,  b2a:.  (a,  |  aj)  |  ((a,  |  a.)  t  o.i)  |  (n,  |  a^,)  )  (((a,  I  a-)  t  a,)  (  (n,  1  a-.)).  Iln 

and  Tucker  [4]  proved  that  the  sum  of  the  first  k  subexpressions  formed  hy  I  lnlTiii.iii’s  algonlhm  i.s  les.s  than 

or  c<|ual  to  the  sum  of  any  k  subexpressions  built  up  successively  starling  with  , .  .  . ,  a„ }  ami  n'plaring 

a,  and  a^  by  a,  |  n,.  (ilas.sey  and  Karp  |lj  sinmed  that  this  ha.s  extensive  roii.seipiences,  for  it  implies  that 
HulTinan’s  algorithm  minimizes  not  only  Ei  v,  , .  i)  I'”'  •'"i)'  uoiidecre.i.-ing  concave 

function  J .  The.se  facl.s  can  be  pul  into  onr  algebraic  framework  in  the  following  way. 
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Let  •  be  an  associative,  commutative  operator  over  A,  satisfying  tlie  following  operations; 

Bl .  (a  o  6)  •  (c  o  cf)  ^  (a  o  c)  *  (A  o  ci). 

B2.  If  a  <  6  then  a  •  c  <  6  •  c. 

B3.  If  o  <  c  then  (a  o  A)  •  c  <  a  •  (A  o  c). 

We  can  now  sh'>w,  by  mimicking  the  previous  proof  in  a  straightforward  way,  that  llulTman's  procedure  has 
the  following  strong  property.  Suppose  that  the  lirst  k  st<-ps  t>f  llurfman's  algorithm  have  reduced  the  initial 
elements  {fij , . , . ,  n ., )  to  the  elements  {n',,...,a',  ^}.  and  consider  any  other  i-sii-p  process  that  o*  *ains 

^.}  by  repeatetlly  choosing  two  elements  {a,, a.}  and  replacing  them  by  (it,  ‘’Hj)-  Then 

<*'i  •  V  <  «"•  ••  ■*<  i  • 

'I'his  generalises  our  previous  result,  which  was  the  special  case  k  n  1.  The  lemma  of  Hu  and  I'ucker 

follows  by  defining  a  o  b  as  in  (1)  and  taking  (u,  n')  •  (A,  6')  (a  |  A,  n'  1  A').  We  ran  let  n  •  A  a  \  h  when 

a  o  A  has  the  form  (4). 

It  is  easy  to  see  that  the  sulx-xpressions  prrxluced  by  lliiirman's  algorithm  ari'  nomlerreasing:  If  wi' 
number  the  .s,'s  in  the  order  they  are  created,  we  have  S|  ^  ■  s,  .laii  van  l.eeiiwi-n  dllj  li;is 

exploited  this  >,o  show  that  llulTman's  procedure  ran  be  carried  out  m  linear  time,  if  we  .i.s.siime  that  the 

inputs  are  given  in  order  n,  a,,  -.  After  k  steps,  the  remaining  ri  A.  elements  will  be  {<i  ,  ,  ,  ,  .  ,u,  } 

and  {s,,.,.,Sc},  for  some  i  <  n  and  j  <  fc;  initially  i  j  k  0.  Then  the  smalle.st  remaining  I'lement 
is  S|,  if  1  n,  otherwise  it  is  min(a,  ^  i.s,);  if  it  is  |  i ,  we  increase  i  by  1,  otherwise  we  mcn-ase  ;  by  1. 
The  serond-stnallosl  eleinenl  is  then  fonnd  in  the  same  way,  using  a,  ,  ,  if  ;  '•  k.  I'ni.illy  s,  ,  ,  is  computed 
and  k  is  increased  by  1. 

Consider  the  behavior  of  this  procedure  in  the  case  of  operation  (II),  wlnm  n,  •-  a,  and 

I  <  <i[  '.v,  2,  Then  s'  >  '2,  so  the  comparison  of  (a,  |  i,a(  ,  ,.n",  ,)  to  {,s  ,.s'  ,  s",.s"')  can  be  based 

entirely  on  the  first  components  a,  j.  i  and  .s,,  where  we  regard  a,  ,  ,  as  smaller  than  .s  in  case  of  ei|nalily. 
The  p.irticnlar  values  of  a',,  it'' ,  a"'  have  no  effect  on  the  algorithm.  It  follows  th.it  v.m  l.eeuwen’s  ellicient 
procedure  can  bo  used  on  the  singleton  elements  rt|  <  ■  ■  ■  'v  n„  insle.ui  of  the  i|ii,uli uples  of  (11),  with  the 

tie-breaking  rule  that  <i  |  j  should  be  prc-ferreil  to  .s,  in  ca.se  of  eipiallly,  we  obt.uii  bm.iiy  Iri-e  (i  e  ,  an 

expression)  that  minimizes  Vu./,.  Kiirtherinore,  among  all  binary  tri'es  ili.il  obtain  the  mmimiiin  of  ii.l  , 
Ibis  oae  also  minimi/a’.s  '^lii',i,  for  all  choices  I  v  ii(  2.  (  I'lie  special  case  I  for  all  i  was  proved  by 

Schwartz  in  j9j.)  This  .same  binary  tree  also  attains  the  lexicograpliic  niinmmm  of 

beran.se  van  I.eeuwen's  algorithm  will  produce  (he  i<l<-ntical  tree  when  operation  (1)  is  used  with  i  1  I  «, 
for  all  siill'ciently  small  (,  and  wi‘  have  }Ju,(l  I  <)'•  cJ  u,  |  Vn  /,  (  )\i,(!;)  |  •  . 
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